Alliance for Systems Change 

— Mid-South 

~ REGIONAL RESOURCE CENTER 

Helping Agfndn .Ma*/ a l>(ff'trrncf 


State 


Systems 

Improvement 

Self-Assessment 


October 2008 


Mid-South Regional Resource Center 
State Systems Improvement 
Self Assessment and Planning Guide 


Overview and Purpose of Document 

This document is designed to be used as an assessment of State systems by State Part 
B and Part C staff and their stakeholders. It provides a detailed process for State 
Education Agencies (SEA) and Lead Agencies (LA) to follow that will guide 
improvement efforts relative to the APR indicators and other statewide initiatives. Its 
purpose is to assist States in 

1 . implementing a systematic approach to designing and refining improvement 
activities, 

2. aligning improvement activities with State priorities, 

3. allocating funding and human resources to State improvement efforts which are 
most effective, and 

4. implementing, tracking, documenting and evaluating improvement efforts 
designed to achieve or maintain compliance and improve performance that lead 
to better results. 

The framework comprises three major components and their related subcomponents 
and practices, including: 

1 . Collection and Analysis of Data 

a. Review/Revise Existing Data Systems 

b. Analyze Data 

c. Clarify Needs for Improvement 

2. Strategic Design of Improvement Activities 

a. Review/Revise Existing State Performance Plan (SPP) 
Improvement Activities 

b. Select and Design New Improvement Activities 

c. Develop Plan of Action for Implementing Improvement Activities 

3. Implementation, Tracking and Documentation 

a. Implement Improvement Activities 

b. Track and Document Implementation of Improvement Activities 

Woven throughout the framework are evaluation practices to ensure SEA and LA staff 
have the information needed to determine the effectiveness of their improvement efforts 
and enhance their systems and processes. 
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Directions for Completing the Systems Improvement Self-Assessment 

For the purposes of this self-assessment and planning process, our adopted definition 
of system refers to interacting, interrelated, or interdependent elements that form a 
complex whole. The practices are intended to be viewed as part of a larger structure. 
They should not be considered solely as isolated steps relative to an indicator, or the 
agency may risk designing an inefficient system with a fragmented approach. 

To guide State agencies in examining their current practices that comprise the systems, 
it is suggested that SEA and LA SPP/APR teams begin with a thoughtful discussion 
about each component and its related subcomponents and practices. This will help the 
team develop a shared understanding of the practices that we believe contribute to 
effective systems that support continuous improvement. 

It is emphasized that a key to this examination of existing systems is to identify 
immediate and high priority changes that will likely improve single or multiple levels of 
systems functioning relative to data, personnel, funding, technical assistance service 
delivery and monitoring systems. A SEA or LA has flexibility in how they approach the 
completion of this self-assessment. One option may be for the SPP/APR team to use 
their collective knowledge of the practices relative to each indicator to discuss how the 
status of these practices relate to a larger system, followed by reaching consensus on 
(1) whether each practice is standard; (2) whether a high priority exists that will require 
planning an immediate follow-up action or set of actions; or (3) whether or not 
improvement of the practice should be revisited during the agency’s next planning cycle. 

Another option the SEA or LA may choose in responding to the self assessment is to 
begin with a focus on a single or set of related priority indicator(s). Decisions relative to 
the indicators are made whether or not a practice is considered standard practice, 
whether a high priority exists and requires immediate follow-up action, or if the practice 
should be revisited during the next planning cycle. 

The intent of conducting this self-assessment and planning process is for States to 
convene stakeholders (internal and external to the SEA and LA, as deemed 
appropriate) to holistically assess their systems. As priority actions are designed, it is 
critical that State systems in their entirety are considered. To this end, the status of 
each practice needs to be determined and the agency should prioritize follow-up actions 
based on what is most important and requires the most immediate attention. 

Development of a work plan that will guide implementation of these decisions is critical. 
The plan should specify the personnel and fiscal resources that will support each action 
and the projected timelines for initiation and completion. To strive toward continuous 
improvement, States are encouraged to monitor implementation of their plan and revisit 
their practices at regular intervals to determine if the systems are supporting 
improvement and producing the desired results. 
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Documenting Responses to the Self-Assessment 

Please insert the descriptive text in the appropriate columns for each practice as 
suggested below: 

Standard Practice: Y/N: Indicate” Y” for Yes or “N” for No if it is considered or not 
considered standard practice of the agency. 

High Priority; Immediate Follow-Up Action: Describe the action step(s) the agency 
will immediately undertake in an effort to improve the practice and ultimately, the 
systems. 

Revisit Next Planning Cycle (mo/yr): If the agency determines that a practice is not 
standard and needs improvement, but is not a high priority at this time, indicate the 
month and year the discussion will be revisited to determine need for follow-up actions. 
The planning cycle refers to any time throughout the year the State typically conducts 
planning activities. 
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TA Delivery 



Component 1: Collection and Analysis of Data 

Description: The Collection and Analysis of Data component includes the protocols and systems of data collection, access, 
analysis, and use. The data system is integral to making informed decisions regarding revising existing improvement activities and 
designing new improvement efforts, including understanding contexts and resources, making logical linkages between issues and 
activities, and involving stakeholders to ensure interconnections with other improvement initiatives. 

A. Subcomponent: Review/Revise Existing Data Systems 

Practice 

Standard 

Practice 

Y/N 

High Priority; Immediate 
Follow-Up Action 

Revisit 

Next 

Planning 

Cycle 

(mo/yr) 

1 . Periodically review data system capabilities, including data elements 
collected, adequacy and timeliness of reporting functions, verification 
of data, and reliability of data entry. 




2. Refine systems for collecting and retrieving data on all indicators, 
as needed, including securing and synthesizing data from all essential 
data sources. 




3. Refine data system to improve capability of extracting and querying 
data across indicators. 




4. Establish/refine protocols for how, when and where to access all 
relevant data (internal and external to the SEA or LA) for each 
indicator. 




5. Create/update interagency agreements to support data sharing. 




B. Subcomponent: Analyze Data 

Practice 

Standard 

Practice 

Y/N 

High Priority; Immediate 
Follow-Up Action 

Revisit 

Next 

Planning 

Cycle 

(mo/yr) 

1 . Inform SPP/APR team members of availability of data for review 
(e.g. who/how/when to access all data). 




2. Identify/convene SEA or LA staff assigned to the SPP/APR team 
and appropriate stakeholders to examine the data. 
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3. Determine gaps between actual performance and State target. 




4. Analyze all available, relevant data for the indicator and conduct a 
root cause analysis. 




5. Verify improvement activities are implemented as intended and 

are consistently implemented. 




6. Analyze evaluation of improvement activity data. Determine how/if 
each activity has contributed to achieving the desired outcome and 
whether the activity needs to be modified. 





Component 2: Strategic Design of Improvement Activities 

Description: The Strategic Design component describes the design of state level initiatives ranging from selection of the specific 
strategies, to the assignment of resources/staff, alignment of improvement structures, evaluation measures, and reporting 
mechanisms. 

A. Subcomponent: Review/Revise Existing SPP improvement Activities 

Practice 

Standard 

Practice 

Y/N 

High Priority; Immediate 
Follow-Up Action 

Revisit 

Next 

Planning 

Cycle 

(mo/yr) 

1 . Use results from root cause analysis to drive decisions 

regarding which activities should be continued, revised or 
eliminated. 




2. Establish clear and logical linkages between root causes 
identified through data analysis and the improvement activity 
selected. 




3. Redesign activities to reflect evidence-based practices. 

Consider information gathered from other states with similar issues 
and contexts as well as guidance from national TA centers or both. 




4. Determine priority order of activities based on research, linkages 
to common needs across indicators, available resources, and 
expected impact on performance. 




5. Design specific action steps that will be implemented to complete 
the activity. 
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6. Check to see that improvement activities are interconnected 

across indicators to the maximum extent appropriate. 




7. Verify alignment of each existing activity with the appropriate 
indicator(s) and confirm that activities collectively cover the six year 
span of the SPP. 




8. Ensure that each existing activity has a measure of evaluation. 




9. Develop and implement methods for measuring impact of each 
activity/set of activities. 




1 0. Dedicate sufficient resources (e.g. people, time and fiscal) 
needed to support the implementation and evaluation of each 
activity. 




1 1 . Strategically select the partners (e g. internal and external to 
agency, such as, Head Start, IHEs, Child Care, State Advisory 
Panels and Interagency Coordinating Councils) who will play a role 
in the improvement activity implementation and identify specifically 
how they will be engaged. 




1 2. Assign indicator responsibilities to SEA/LA staff for serving as 
lead in obtaining the data for analysis. Adjust staff FTE as needed. 




13. Confirm and document implementers, leaders, and timelines for 

implementing and completing each activity. 




14. Consider vital connections. Ensure linkage with State priorities 
(e.g. State Board of Education, legislative) and all State 
improvement efforts. 




1 5. Assess the State and regional TA service delivery system to 

determine sufficiency to support implementation of improvement 
activities. Make adjustments as needed. 




1 6. Review/revise the funding mechanisms to support all levels of 
implementation of the SPP. Fund new initiatives as needed. 
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B. Subcomponent: Select and Design New Improvement Activities 

Practice 

Standard 

Practice 

Y/N 

High Priority; Immediate 
Follow-Up Action 

Revisit 

Next 

Planning 

Cycle 

(mo/yr) 

1 . Invite stakeholders to review data with SEA/LA staff to 

determine any gaps identified between State target and 
performance. Engage staff from other divisions within the LA/SEA 
and other State initiative leads to gain broader perspectives. 




2. Identify indicator areas needing improvement and use results 
from root cause analysis to determine whether a new activity needs 
to be designed to address the specified need. 




3. Determine what success will look like by identifying what you 
expect to see or hear that is different after implementing an 
improvement activity. 




4. Design initial activities that create a foundation from which other 
activities may be built. 




5. Confirm that each activity is: 

a. Desirable: SEA or LA considers the activity to be necessary and 
wants to do the activity 

b. Feasible: the activity is viewed as realistic to accomplish in the 
timelines reported and the number of activities per indicator(s) is 
realistic. Consider whether staff and funding are available and 
any related circumstances that are unique to the State. 

c. Motivating: willingness of SEA or LA to obligate the people and 
fiscal resources, commitment of staff and partners to work to 
make it happen 




6. Determine priority order of new and existing activities based on 
research, linkages to common needs across indicators, available 
resources, and expected impact on performance. 




7. Sequentially order activities that demonstrate a progression from 
simple to complex actions. 
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8. Identify evaluation questions SEA or LA needs to answer for 
each single activity or set of activities to improve the data. 




9. Design evaluation measure(s) and method(s) of evaluation for 
each activity and/or set of activities. Consider baseline measures, 
intermediate (formative) measures to check progress, and 
summative measures to confirm improvement outcomes. 




1 0. Design approaches to promote improvement through 
incentives (e.g. protocols for determination of 
program/district/school eligibility for and use of discretionary funds 
or resources, system for allocating funds, strategies for public 
recognition of program/district/school accomplishments and models 
of excellence). 




1 1 . Develop infrastructures to promote improvement through 

capacity building (e.g. structures for determining improvements in 
skills and knowledge and processes for developing and supporting 
implementation of improvement plans). 




12. Develop communication practices for programs/schools/districts 
to promote improvement through providing opportunities for change: 

• methods for announcing available services and opportunities; 

• protocols for accessing waiver processes and alternate 
routes to certification; 

• methods for disseminating evidence-based “how to” 
information; 

• models for demonstrations and pilot projects 




1 3. Assign a measurement that directly measures the result(s) of the 
activity. 




1 4. Select method and identify staff responsible for conducting the 
evaluation. 
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1 5. Determine data source for each activity that will be used to assess 
the process and impact of the activity. 




1 6. Assign implementers for each activity (and each action, if different 
staff required). Consider staff connected with other state priorities 
and initiatives (internal and external to the SEA/LA). 




1 7. Develop higher leverage activities that are built from 
foundational activities. 




1 8. Review/revise the sequence of activities (existing and proposed) 
for a single indicator or set of indicators to ensure they are logically 
organized and prioritized. 




1 9. Align or consolidate the improvement activities with general 
education initiatives (e.g. preschool standards and curricula, 
NCLB, assessment and school improvement) and statewide 
systems change initiatives and programs(e.g. ICC/SAP 
goals/plans, State Board of Education goals or State Plans, State 
Legislative priorities, SPDG, Expanding Opportunities). 




20. Coordinate all relevant TA activities with general education TA 
delivery systems. 




21 . Ensure that all subcontractors (e g. SPDG, TA providers) align 
their work with the improvement activities. Install a system to 
obtain their data to measure process and impact of improvement 
efforts. 




22 . Assign fiscal and staff resources needed to support 
implementation and evaluation. 




23. Disseminate information to regional and local TA providers on 

the most current guidance related to the indicators. 




24. Coordinate improvement activities with local program 
processes and timelines (e.g., calendars, self-assessment, 
planning, reporting, and evaluation). 




25. Design new activities, as needed, which are expected to have the 
most significant impact on improving the data. 
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C. Subcomponent: Develop Plan of Action for Implementing 

Improvement Activities 

Practice 

Standard 

Practice 

Y/N 

High Priority; Immediate 
Follow-Up Action 

Revisit 

Next 

Planning 

Cycle 

(mo/yr) 

1 . Design the specific action steps that must be carried out to 
accomplish the activity. 




2. Identify/document the agency/organization and role of staff 

(e.g. State TA system, monitoring coordinator, assessment 
consultant, SPDG director) responsible for carrying out each 
action. 




3. Designate timelines for implementation of each action step. 

Compare timelines across indicators to determine if all are 

feasible to accomplish within the given timeframe. 




4. Confirm expectations with implementers for executing each set 
of actions. 




5. Monitor and adjust action steps as needed to fully accomplish the 
activity. 
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Component 3: Implementation, Tracking and Documentation 

Description: The Implementation, Tracking and Documentation component includes the implementation of the improvement 
activities, including data systems, training/professional development, procedural/policy changes, monitoring, personnel assignments, 
program development, collaboration/coordination, and other initiatives designed to achieve or maintain compliance and improve 
results. 

A. Subcomponent: Implement improvement activities. 

Practice 

Standard 

Practice 

Y/N 

High Priority; Immediate 
Follow-Up Action 

Revisit 

Next 

Planning 

Cycle 

(mo/yr) 

1 . Execute all actions outlined for each improvement activity. 




2. Maintain ongoing communications to discuss progress between 
each activity lead and SEA/LA staff responsible for ensuring activity 
implementation. 




3. Gauge inputs and outputs as plan is implemented to assess 
progress, adherence to timelines, and access to resources. 




4. Evaluate fidelity of implementation of each activity. 




5. Make mid-course adjustments as needed (e.g. if implementers 
need additional support to do what they agreed to do). 




6. Review and refine activities as needed to meet the target. 




B. Subcomponent: Track and document implementation of 

mprovement activities. 

Practice 

Standard 

Practice 

Y/N 

High Priority; Immediate 
Follow-Up Action 

Revisit 

Next 

Planning 

Cycle 

(mo/yr) 

1 . Identify SEA or LA staff leads to track implementation (actions 
and results) of improvement activities for each indicator or set of 
indicators. 




2. Hold regularly scheduled meetings of SPP/APR Team (and 
implementers of activities if they are external to the Team) to 
discuss progress, concerns, TA needs and ongoing opportunities for 
partnering across indicators/activities. 
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3. Develop new databases/templates or utilize existing processes 
for entering and tracking status and progress of implementation of 
activities (e.g. documenting discussions relative to actions taken, 
adherence to timelines, results of activities, and needed follow-up 
actions and decisions). 




4. Utilize existing / create new system of evaluating process and 
impact of activities. 




5. Seek data and maintain documentation on implementation from 
all sources that support each improvement activity (e.g., SPDG, 
regional TA entities and subcontractors). 




6. Track all resources dedicated to TA delivery focused on 
improving data in the SPP/APR indicators. 




7. Monitor and evaluate effectiveness of TA services delivered in 
improving the data. 




8. Pinpoint any unintended outcomes. Determine how this 

information will be used to inform decision-making regarding current 
and future improvement efforts. 




9. Document and share systemic change results with El programs, 
LEAs and other stakeholders. 




1 0. Continue to collect data and develop root cause analysis on 

improvement efforts for each indicator. Use results of root cause 
analysis to drive decisions on an annual basis regarding which 
activities should be continued, revised or eliminated. 




1 1 . Dedicate human and fiscal resources to support implementation 
of evaluation plans. 




1 2. Conduct formative evaluation to guide when and type of mid- 
course adjustments needed. 




1 3. Conduct impact evaluation Determine extent to which activity/set 
of activities caused a desired change (results focused); 

Confirm that activity helped State meet their target, achieve or 
maintain compliance or improve performance. If activity was not 
deemed successful, analyze the reasons, assess whether any 
aspect of the activity influenced the desired outcome not being 





Mid-South Regional Resource Center, October 2008 


13 




achieved and use these discoveries to inform maintenance or re- 
design of future improvement activities. 




1 4. Continue to evaluate and refine systems and infrastructures of 

technical assistance and support, data collection and reporting, 
SEA/LA systems administration, monitoring, and resource 
allocations. 
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